fix(stack): two k3s blockers — loopback Endpoints, and the storefront preview host missing from /etc/hosts - #808
Conversation
`obol stack up` fails on the k3s backend with: UPGRADE FAILED: cannot patch "ollama" with kind Endpoints: Endpoints "ollama" is invalid: subsets[0].addresses[0].ip: Invalid value: "127.0.0.1": may not be in the loopback range k3s runs directly on the host, so OllamaHostForBackend returns 127.0.0.1 and that value is stamped straight into the ollama Endpoints. Kubernetes has rejected loopback addresses in Endpoints since v1.33, and rightly so: inside a pod's network namespace 127.0.0.1 is the pod itself, not the host, so the endpoint could never have routed anywhere useful. OllamaHostIPForBackend now substitutes the host's primary routable IPv4 address whenever resolution lands on loopback. The guard sits in the shared resolver rather than in a k3s branch, so it also covers Docker runtimes that map host.docker.internal to loopback. resolveHostIP in `obol sell` was a second copy of the same strategy and had drifted with the same bug — it returned 127.0.0.1 for k3s and fed it to createHostService, which builds an Endpoints object too. It now delegates to the shared resolver instead of duplicating it. Three existing tests asserted the old loopback values; they now assert the contract that actually matters — the resolved address is a valid, non-loopback IP.
The /storefront branding editor iframes storefront-preview.obol.stack, so that name has to resolve locally. `obol stack up` appended it to the hostname list at one call site — but EnsureHostsEntries replaces the managed /etc/hosts block wholesale, and the four other call sites in internal/hermes and internal/openclaw pass only agent hostnames. `stack up` resumes agents after syncing defaults, so the hermes path runs last and rewrites the block without the preview origin. Observed on a real k3s stack: the block ended up with obol.stack and the two agent hosts, the preview name did not resolve, and the editor's iframe had nowhere to load from — while the HTTPRoute and its backend were healthy and answered fine on a Host header. The origin is a fixed property of every local stack, exactly like the base domain, so it is now emitted unconditionally rather than depending on which caller happens to write last. The constant moves to internal/dns (a leaf package that owns the managed block) and internal/tunnel points at it, keeping one source of truth. Block rendering is split into buildHostsBlock so the guarantee is testable without root.
bussyjd
changed the title
Second commit added — and the honest status of the firstI was asked whether the loopback fix was actually validated. It wasn't: it was unit-tested on macOS only. Validating it properly on a real k3s box surfaced a second, independent blocker, so this PR now carries two commits. Validation trap worth recordingMy first attempt to validate on a k3s host appeared to fail — The loopback fix changes Go code, not template bytes, and I had deliberately built with The second bugWith Cause:
Fixed by emitting the origin unconditionally, like the base domain, instead of relying on call-site discipline. The constant moves to End-to-end on a k3s host — server v1.35.5+k3s1Ollama on that host listens on The
Still not covered
|
1 participant
The failure
obol stack upfails on the k3s backend, blocking the whole defaults helmfile:Reproduced on a k3s host (server
v1.35.5+k3s1) while validatingv0.14.0-rc2. It is not a regression from that RC — the same box's helm history shows it failing identically under0.13.0-rc3four weeks earlier:Why it happens
OllamaHostForBackendreturns127.0.0.1for k3s — correct in spirit, since k3s runs directly on the host.OllamaHostIPForBackendsees a value that already parses as an IP and returns it unchanged, and it lands verbatim in theollamaEndpoints inbase/templates/llm.yaml.Kubernetes has rejected loopback addresses in Endpoints since v1.33, and it is right to: inside a pod's network namespace
127.0.0.1is the pod, not the host. The endpoint could never have routed to host Ollama — the API server now simply refuses to pretend otherwise. Any k3s user on Kubernetes ≥1.33 hits this.The fix
One guard in the shared resolver: when resolution lands on loopback, substitute the host's primary routable IPv4 address. Deliberately not a k3s branch — it also covers Docker runtimes that map
host.docker.internalto loopback.hostPrimaryIP()asks the kernel which route it would take (a UDP "dial" to TEST-NET-1, no packets sent), and falls back to scanning for the first up, non-loopback IPv4 on hosts with no default route.Sibling caller found and collapsed
Grepping the callers turned up
resolveHostIPincmd/obol/sell.go— a second copy of the same strategy that had drifted with the identical bug: it returned127.0.0.1for k3s and handed it tocreateHostService, which also builds an Endpoints object. Soobol sell inferenceon k3s would fail the same way. It now delegates to the shared resolver instead of duplicating it (net −27 lines there).The stale
createHostServicedoc comment already admitted the duplication: "using the same strategy as ollamaHostIPForBackend in internal/stack".Tests
Three existing tests asserted the old loopback values — they encoded the bug. They now assert the contract that matters: the resolved address is a valid, non-loopback IP.
TestOllamaHostIPForBackend_K3s— was== "127.0.0.1", now "valid and not loopback"TestOllamaHostIPForBackend_AlreadyIP— still exercises the numeric short-circuit path, without pinning the loopback valueTestCopyInfrastructureRendersStackPlaceholders— was a literalip: "127.0.0.1"substring match, now extracts the rendered IP and asserts it is routablePlus two new regression tests:
TestOllamaHostIPForBackendNeverReturnsLoopbackandTestHostPrimaryIPIsRoutable.Verification:
go build ./...OK ·go vet ./...clean ·go test ./...exit 0, 40 packages ·gofmtclean on all touched files.Notes
integration/v0.14.0-rc2so it can go into an rc3 and unblock end-to-end validation of the storefront live preview. It should also land onmain— it affects every k3s user, independent of this release train.baseon any cluster that already has a failed release revision may needhelm rollbackbefore a cleanstack up. That is remediation on existing boxes, not a code fix.